{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "accelerator": "TPU",
    "colab": {
      "name": "W3_Tutorial1.ipynb",
      "provenance": [],
      "collapsed_sections": [],
      "machine_shape": "hm",
      "include_colab_link": true
    },
    "kernelspec": {
      "display_name": "Python 3",
      "name": "python3"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/CIS-522/course-content/blob/main/tutorials/W03_MLPs/W3_Tutorial1.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "X2kPrtVyXtaP"
      },
      "source": [
        "# CIS-522 Week 3 Part 1\r\n",
        "# Multi-Layer Perceptrons (MLPs)\r\n",
        "\r\n",
        "__Instructor__: Konrad Kording\r\n",
        "\r\n",
        "__Content creators:__ Arash Ash"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "p2DuJddRXxYY"
      },
      "source": [
        "---\r\n",
        "# Tutorial objectives\r\n",
        "In this tutorial, we delve deeper by using one of the most famous deep learning models of all!\r\n",
        "\r\n",
        "MLPs are arguably one of the most tractable models that we can use to study deep learning fundamentals. Here we will learn why MLPs are: \r\n",
        "\r\n",
        "* similar to biological networks\r\n",
        "* good at function approximation"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "e6GnDC12UtUD",
        "cellView": "form"
      },
      "source": [
        "#@markdown What is your Pennkey and pod? (text, not numbers, e.g. bfranklin)\n",
        "my_pennkey = 'test' #@param {type:\"string\"}\n",
        "my_pod = 'Select' #@param ['Select', 'euclidean-wombat', 'sublime-newt', 'buoyant-unicorn', 'lackadaisical-manatee','indelible-stingray','superfluous-lyrebird','discreet-reindeer','quizzical-goldfish','astute-jellyfish','ubiquitous-cheetah','nonchalant-crocodile','fashionable-lemur','spiffy-eagle','electric-emu','quotidian-lion']\n"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "cellView": "form",
        "id": "aduF74_wPXDQ"
      },
      "source": [
        "# @title Slides\r\n",
        "from IPython.display import HTML\r\n",
        "HTML('<iframe src=\"https://docs.google.com/presentation/d/e/2PACX-1vSPvHqDTmMq4GyQy6lieNEFxq4qz1SmqC2RNoeei3_niECH53zneh8jJVYOnBIdk0Uaz7y2b9DK8V1t/embed?start=false&loop=false&delayms=3000\" frameborder=\"0\" width=\"480\" height=\"299\" allowfullscreen=\"true\" mozallowfullscreen=\"true\" webkitallowfullscreen=\"true\"></iframe>')"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "np4EZarUtKoA"
      },
      "source": [
        "# Recap the experience from last week\r\n",
        "\r\n",
        "We focused on linear deep learning last week. We discussed Artificial Neural Networks and saw how it works, the dynamics of learning, and the properties of high dimensional spaces. You should now have some intuition about deep learning systems we will learn. We also dived into PyTorch and autograd, which are tools that make our life easy. "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "wEdRnCm2_DT0",
        "cellView": "form"
      },
      "source": [
        "#@title Video: Discussing Week 2 - Linear DL\n",
        "import time\n",
        "try: t0;\n",
        "except NameError: t0=time.time()\n",
        "\n",
        "from IPython.display import YouTubeVideo\n",
        "\n",
        "video = YouTubeVideo(id=\"xlYttP5C_LY\", width=854, height=480, fs=1)\n",
        "print(\"Video available at https://youtube.com/watch?v=\" + video.id)\n",
        "\n",
        "video"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "E29pz3hNeQTO"
      },
      "source": [
        "Meet with your pod for 10 minutes to discuss what you learned, what was clear, and what you hope to learn more about."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "c8nOPFKTuJhA",
        "cellView": "form"
      },
      "source": [
        "#@markdown Tell us your thoughts about what you have learned in week 2.\n",
        "w2_upshot = '' #@param {type:\"string\"}"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Bh1hjT8_Wleu"
      },
      "source": [
        "# Question of the week"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "cellView": "form",
        "id": "tdJuZdmMWhGw"
      },
      "source": [
        "#@markdown What functional forms are good or bad for representing complex functions?\r\n",
        "w3_q = '' #@param {type:\"string\"}"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "vgOQxVS1X2dB"
      },
      "source": [
        "---\r\n",
        "# Setup"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "xXd8Now_XwR7"
      },
      "source": [
        "# imports\r\n",
        "import random\r\n",
        "import pathlib\r\n",
        "\r\n",
        "import torch\r\n",
        "import numpy as np\r\n",
        "import matplotlib.pyplot as plt\r\n",
        "\r\n",
        "import torch.nn as nn\r\n",
        "import torch.optim as optim\r\n",
        "import torch.nn.functional as F\r\n",
        "import torchvision.transforms as transforms\r\n",
        "from torchvision.datasets import ImageFolder\r\n",
        "from torch.utils.data import DataLoader, TensorDataset\r\n",
        "from torchvision.utils import make_grid\r\n",
        "from IPython.display import HTML, display\r\n",
        "\r\n",
        "# Make sure the Runtime is with TPU to ensure maximum speed\r\n",
        "dev = \"cpu\""
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "ZUyEQpxTSzxe",
        "cellView": "form"
      },
      "source": [
        "# @title Seeding for reproducibility\r\n",
        "seed = 522\r\n",
        "random.seed(seed)\r\n",
        "np.random.seed(seed)\r\n",
        "torch.manual_seed(seed)\r\n",
        "torch.cuda.manual_seed(seed)\r\n",
        "\r\n",
        "torch.backends.cudnn.deterministic = True\r\n",
        "torch.backends.cudnn.benchmark = False\r\n",
        "torch.set_deterministic(True)\r\n",
        "def seed_worker(worker_id):\r\n",
        "    worker_seed = seed % (worker_id+1)\r\n",
        "    np.random.seed(worker_seed)\r\n",
        "    random.seed(worker_seed)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "hj8C2Cf0G0i_",
        "cellView": "form"
      },
      "source": [
        "# @title Dataset download\r\n",
        "%%capture\r\n",
        "!rm -r AnimalFaces32x32/\r\n",
        "!git clone https://github.com/arashash/AnimalFaces32x32\r\n",
        "!rm -r afhq/\r\n",
        "!unzip ./AnimalFaces32x32/afhq_32x32.zip "
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "oRrTcZOlXA8C",
        "cellView": "form"
      },
      "source": [
        "# @title Figure settings\r\n",
        "import ipywidgets as widgets\r\n",
        "%matplotlib inline \r\n",
        "fig_w, fig_h = (8, 6)\r\n",
        "plt.rcParams.update({'figure.figsize': (fig_w, fig_h)})\r\n",
        "%config InlineBackend.figure_format = 'retina'\r\n",
        "my_layout = widgets.Layout()"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "40Qc7MFcX7Oc",
        "cellView": "form"
      },
      "source": [
        "# @title Helper functions\r\n",
        "def imshow(img):\r\n",
        "    img = img / 2 + 0.5     # unnormalize\r\n",
        "    npimg = img.numpy()\r\n",
        "    plt.imshow(np.transpose(npimg, (1, 2, 0)))\r\n",
        "    plt.axis(False)\r\n",
        "    plt.show()\r\n",
        "\r\n",
        "def progress(epoch, loss, epochs=100):\r\n",
        "    return HTML(\"\"\"\r\n",
        "        <label for=\"file\">Training loss: {loss}</label>\r\n",
        "        <progress\r\n",
        "            value='{epoch}'\r\n",
        "            max='{epochs}',\r\n",
        "            style='width: 100%'\r\n",
        "        >\r\n",
        "            {epoch}\r\n",
        "        </progress>\r\n",
        "    \"\"\".format(loss=loss, epoch=epoch, epochs=epochs))\r\n",
        "\r\n",
        "def plot_function_approximation(x, relu_acts, y_hat):\r\n",
        "  fig, axes = plt.subplots(2, 1)\r\n",
        "\r\n",
        "  # Plot ReLU Activations\r\n",
        "  axes[0].plot(x, relu_acts.T);\r\n",
        "  axes[0].set(xlabel = 'x', ylabel = 'Activation', title = 'ReLU Activations')\r\n",
        "  labels = [f'ReLU {i + 1}' for i in range(relu_acts.shape[0])]\r\n",
        "  axes[0].legend(labels, ncol = 2)\r\n",
        "\r\n",
        "  # Plot function approximation\r\n",
        "  axes[1].plot(x, torch.sin(x), label = 'truth')\r\n",
        "  axes[1].plot(x, y_hat, label = 'estimated')\r\n",
        "  axes[1].legend()\r\n",
        "  axes[1].set(xlabel = 'x', ylabel = 'y(x)', title = 'Function Approximation');\r\n",
        "\r\n",
        "  plt.tight_layout()\r\n",
        "  plt.show()"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "UQGBDFB0yeqV"
      },
      "source": [
        "---\r\n",
        "# Section 1: Neuron Physiology"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "FNdeptL-zUnU",
        "cellView": "form"
      },
      "source": [
        "#@title Video: Overview and Integrate-and-Fire Neurons\n",
        "try: t1;\n",
        "except NameError: t1=time.time()\n",
        "\n",
        "video = YouTubeVideo(id=\"exTzHGfEAvU\", width=854, height=480, fs=1)\n",
        "print(\"Video available at https://youtube.com/watch?v=\" + video.id)\n",
        "\n",
        "video"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "c93MxeKdfGN7"
      },
      "source": [
        "## Section 1.1: Leaky Integrate-and-fire (LIF)\r\n",
        "The basic idea of LIF neuron was proposed in 1907 by Louis Édouard Lapicque, long before we understood the electrophysiology of a neuron (see a translation of [Lapicque's paper](https://pubmed.ncbi.nlm.nih.gov/17968583/) ). More details of the model can be found in the book [**Theoretical neuroscience**](http://www.gatsby.ucl.ac.uk/~dayan/book/) by Peter Dayan and Laurence F. Abbott.\r\n",
        "\r\n",
        "The model dynamics is defined with the following formula,\r\n",
        "\r\n",
        "$$\r\n",
        "\\frac{d V}{d t}=\\left\\{\\begin{array}{cc}\r\n",
        "\\frac{1}{C}\\left(-\\frac{V}{R}+I \\right) & t>t_{r e s t} \\\\\r\n",
        "0 & \\text { otherwise }\r\n",
        "\\end{array}\\right.\r\n",
        "$$\r\n",
        "\r\n",
        "Note that $V$, $C$, and $R$ are the membrane voltage, capacitance, and resitance of the neuron respectively and $-\\frac{V}{R}$ is the leakage current. When $I$ is sufficiently strong such that $V$ reaches a certain threshold value $V_{\\rm th}$, it momentarily spikes and then $V$ is reset to $V_{\\rm reset}< V_{\\rm th}$, and voltage stays at $V_{\\rm reset}$ for $\\tau_{\\rm ref}$ ms, mimicking the refractoriness of the neuron during an action potential (note that $V_{\\rm reset}$ and $\\tau_{\\rm ref}$ is assumed to be zero in the lecture):\r\n",
        "\r\n",
        "\\begin{eqnarray}\r\n",
        "V(t)=V_{\\rm reset} \\text{  for } t\\in(t_{\\text{sp}}, t_{\\text{sp}} + \\tau_{\\text{ref}}]\r\n",
        "\\end{eqnarray}\r\n",
        "\r\n",
        "where $t_{\\rm sp}$ is the spike time when $V(t)$ just exceeded $V_{\\rm th}$.\r\n",
        "\r\n",
        "Thus, the LIF model captures the facts that a neuron:\r\n",
        "- performs spatial and temporal integration of synaptic inputs \r\n",
        "- generates a spike when the voltage reaches a certain threshold\r\n",
        "- goes refractory during the action potential\r\n",
        "- has a leaky membrane \r\n",
        "\r\n",
        "For in-depth content on computational models of neurons, follow the [NMA](https://www.neuromatchacademy.org/) Week 3 Day 1 material on Real Neurons and specifically this [Tutorial](https://colab.research.google.com/github/NeuromatchAcademy/course-content/blob/master/tutorials/W3D1_RealNeurons/W3D1_Tutorial1.ipynb).\r\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "fW9mD1wC11Kw"
      },
      "source": [
        "## Exercise 1: Simulating an LIF Neuron\r\n",
        "\r\n",
        "In the cell below is given a function for LIF neuron model with it's arguments described. Now, it's your turn to try this simple mathematical model of a neuron:\r\n",
        "\r\n",
        "Note that we will use Euler's method to make a numerical approximation to a derivative. Hence we will use the following implementation of the model dynamics,\r\n",
        "\r\n",
        "$$\r\n",
        "V_n=\\left\\{\\begin{array}{cc}\r\n",
        "V_{n-1} + \\frac{1}{C}\\left(-\\frac{V}{R}+I \\right) \\Delta t & t>t_{r e s t} \\\\\r\n",
        "0 & \\text { otherwise }\r\n",
        "\\end{array}\\right.\r\n",
        "$$"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "bTydKkuOeqVc"
      },
      "source": [
        "def run_LIF(I, T = 50, dt = 0.1, t_rest = 0, tau_ref = 4,\r\n",
        "            Rm = 1, Cm = 10, Vth = 1, V_spike = 0.5):\r\n",
        "  \"\"\"\r\n",
        "  Simulate the LIF dynamics with external input current\r\n",
        "\r\n",
        "  Args:\r\n",
        "    I          : input current (mA)\r\n",
        "    T          : total time to simulate (msec)\r\n",
        "    dt         : simulation time step (msec)\r\n",
        "    t_rest     : initial refractory time\r\n",
        "    tau_ref    : refractory period (msec)\r\n",
        "    Rm         : resistance (kOhm)\r\n",
        "    Cm         : capacitance (uF)\r\n",
        "    Vth        : spike threshold (V)\r\n",
        "    V_spike    : spike delta (V)\r\n",
        "\r\n",
        "  Returns:\r\n",
        "    time       : time points\r\n",
        "    Vm         : membrane potentials\r\n",
        "  \"\"\"\r\n",
        "\r\n",
        "  ####################################################################\r\n",
        "  # Fill in all missing code below (...),\r\n",
        "  # then remove or comment the line below to test your function\r\n",
        "  raise NotImplementedError(\"Define the LIF model\")\r\n",
        "  ####################################################################\r\n",
        "\r\n",
        "  # Set up array of time steps\r\n",
        "  time = torch.arange(0, T+dt, dt)\r\n",
        "  \r\n",
        "  # Set up array for tracking Vm\r\n",
        "  Vm = torch.zeros(len(time)) \r\n",
        "\r\n",
        "  # Iterate over each time step\r\n",
        "  for i, t in enumerate(time):\r\n",
        "\r\n",
        "    # If t is after refractory period\r\n",
        "    if t > t_rest:\r\n",
        "      Vm[i] = ...\r\n",
        "\r\n",
        "    # If Vm is over the threshold\r\n",
        "    if Vm[i] >= Vth:\r\n",
        "      \r\n",
        "      # Increase volatage by change due to spike\r\n",
        "      Vm[i] += ...\r\n",
        "\r\n",
        "      # Set up new refactory period\r\n",
        "      t_rest = ...\r\n",
        "\r\n",
        "  return time, Vm\r\n",
        "\r\n",
        "\r\n",
        "### Uncomment below to test your function\r\n",
        "# sim_time, Vm = run_LIF(1.5)\r\n",
        "\r\n",
        "# plt.plot(sim_time, Vm)\r\n",
        "# plt.title('LIF Neuron Output')\r\n",
        "# plt.ylabel('Membrane Potential (V)')\r\n",
        "# plt.xlabel('Time (msec)')\r\n",
        "# plt.show()"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "FixKcRzqdhf3"
      },
      "source": [
        "# to_remove solution\r\n",
        "\r\n",
        "def run_LIF(I, T = 50, dt = 0.1, t_rest = 0, tau_ref = 4,\r\n",
        "            Rm = 1, Cm = 10, Vth = 1, V_spike = 0.5):\r\n",
        "  \"\"\"\r\n",
        "  Simulate the LIF dynamics with external input current\r\n",
        "\r\n",
        "  Args:\r\n",
        "    I          : input current (mA)\r\n",
        "    T          : total time to simulate (msec)\r\n",
        "    dt         : simulation time step (msec)\r\n",
        "    t_rest     : initial refractory time\r\n",
        "    tau_ref    : refractory period (msec)\r\n",
        "    Rm         : resistance (kOhm)\r\n",
        "    Cm         : capacitance (uF)\r\n",
        "    Vth        : spike threshold (V)\r\n",
        "    V_spike    : spike delta (V)\r\n",
        "\r\n",
        "  Returns:\r\n",
        "    time       : time points\r\n",
        "    Vm         : membrane potentials\r\n",
        "  \"\"\"\r\n",
        "\r\n",
        "  # Set up array of time steps\r\n",
        "  time = torch.arange(0, T+dt, dt) \r\n",
        "  \r\n",
        "  # Set up array for tracking Vm\r\n",
        "  Vm = torch.zeros(len(time)) \r\n",
        "\r\n",
        "  # Iterate over each time step\r\n",
        "  for i, t in enumerate(time):\r\n",
        "\r\n",
        "    # If t is after refractory period\r\n",
        "    if t > t_rest:\r\n",
        "      Vm[i] = Vm[i-1] + 1/Cm*(-Vm[i-1]/Rm + I)  * dt \r\n",
        "\r\n",
        "    # If Vm is over the threshold\r\n",
        "    if Vm[i] >= Vth:\r\n",
        "\r\n",
        "      # Increase volatage by change due to spike\r\n",
        "      Vm[i] += V_spike \r\n",
        "\r\n",
        "      # Set up new refactory period\r\n",
        "      t_rest = t + tau_ref \r\n",
        "\r\n",
        "  return time, Vm\r\n",
        "\r\n",
        "\r\n",
        "### Uncomment below to test your function\r\n",
        "sim_time, Vm = run_LIF(1.5)\r\n",
        "with plt.xkcd():\r\n",
        "  plt.plot(sim_time, Vm)\r\n",
        "  plt.title('LIF Neuron Output')\r\n",
        "  plt.ylabel('Membrane Potential (V)')\r\n",
        "  plt.xlabel('Time (msec)')\r\n",
        "  plt.show()"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "_5hljR2LZXxS"
      },
      "source": [
        "## Section 1.2: Nonlinearity of LIF neurons"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "vh7Wnzg3ZYaY",
        "cellView": "form"
      },
      "source": [
        "#@title Video: Are Integrate-and-Fire Neurons Linear?\n",
        "\n",
        "video = YouTubeVideo(id=\"6IzHZB7xf34\", width=854, height=480, fs=1)\n",
        "print(\"Video available at https://youtube.com/watch?v=\" + video.id)\n",
        "\n",
        "video"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "aw0w47-O6Arl"
      },
      "source": [
        "## Interactive demo: F-I explorer for different $R_m$\r\n",
        "We know that neurons communicate by modulating the spike count. Therefore it makes sense to characterize their spike count as a function of input current. This is called the neuron's input-output transfer function (so simply F-I curve). Let's plot the neuron's F-I curve and see how it changes with respect to the membrane resistance? "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "cellView": "form",
        "id": "MPKolg9xjiHW"
      },
      "source": [
        "# @title\r\n",
        "\r\n",
        "# @markdown Make sure you execute this cell to enable the widget!\r\n",
        "\r\n",
        "@widgets.interact(Rm=widgets.FloatSlider(1., min=0.5, max=10., step=0.1, layout=my_layout))\r\n",
        "\r\n",
        "def plot_IF_curve(Rm):\r\n",
        "  T = 100 # total time to simulate (msec)\r\n",
        "  dt = 1 # simulation time step (msec)\r\n",
        "  Vth = 1 # spike threshold (V)\r\n",
        "  Is = torch.linspace(0, 2, 10)\r\n",
        "  spike_counts = []\r\n",
        "  for I in Is:\r\n",
        "    _, Vm = run_LIF(I, T = T, Vth = Vth, Rm=Rm)\r\n",
        "    spike_counts += [torch.sum(Vm > Vth)]\r\n",
        "\r\n",
        "  plt.plot(Is, spike_counts)\r\n",
        "  plt.title('LIF Transfer Function (I/F Curve)')\r\n",
        "  plt.ylabel('Spike count')\r\n",
        "  plt.xlabel('I (mA)')\r\n",
        "  plt.show()"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "cellView": "form",
        "id": "adkXbLroqeKc"
      },
      "source": [
        "#@markdown Why do you think increasing the membrane resistance leads to a more ReLU like transfer function?\r\n",
        "w3_why_become_relu = '' #@param {type:\"string\"}"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "DkvnWkvTz2O5"
      },
      "source": [
        "---\n",
        "# Section 2: The Need for MLPs"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "tOgsH7ly5ZU1",
        "cellView": "form"
      },
      "source": [
        "#@title Video: The XOR Problem\n",
        "\n",
        "try: t2;\n",
        "except NameError: t2=time.time()\n",
        "\n",
        "video = YouTubeVideo(id=\"PERmPT1cOP0\", width=854, height=480, fs=1)\n",
        "print(\"Video available at https://youtube.com/watch?v=\" + video.id)\n",
        "\n",
        "video"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "hrEdLosLecm5"
      },
      "source": [
        "In the previous tutorial we have illustrated a weird solution to the XOR problems using a linear NN. However, this has happened because of a nonlinear regime in tiny values. Here, we are exposed to a limitation of linear systems, namely in solving the XOR problem. One cannot draw a straight line to separate the positive (1) and negative examples(0). \n",
        "The nonlinearity of the multilayer perceptron comes in handy to solve this problem. \n",
        "We will visualize this concept next: "
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ZJEy_D2A53SV"
      },
      "source": [
        "## Exercise 2: Solving XOR \r\n",
        "Here we use an open source and famous visualization widget developed by Tensorflow team available [here](https://github.com/tensorflow/playground).\r\n",
        "* Play with the widget and observe that you can not solve the continuous XOR dataset\r\n",
        "* Now add one hidden layer with three units, play with the widget, and set weights by hand to solve this dataset perfectly.\r\n",
        "\r\n",
        "For the second part, you should set the weights by clicking on the connections and either type the value or use the up and down keys to change it by one increment. You could also do the same for the biases by clicking on the tiny square to each neuron's bottom left.\r\n",
        "Even though there are infinitely many solutions, a neat solution when $f(x)$ is ReLU is: \r\n",
        "\r\n",
        "$$y = f(x_1)+f(x_2)-f((x_1+x_2))$$\r\n",
        "\r\n",
        "Try to set the weights and biases to implement this function after you played enough :)"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "ybfxBilvrWH_",
        "cellView": "form"
      },
      "source": [
        "# @title XOR Exercise\r\n",
        "from IPython.display import HTML\r\n",
        "HTML('<iframe width=\"1020\" height=\"660\" src=\"https://playground.arashash.com/#activation=relu&batchSize=10&dataset=xor&regDataset=reg-plane&learningRate=0.03&regularizationRate=0&noise=0&networkShape=&seed=0.91390&showTestData=false&discretize=false&percTrainData=90&x=true&y=true&xTimesY=false&xSquared=false&ySquared=false&cosX=false&sinX=false&cosY=false&sinY=false&collectStats=false&problem=classification&initZero=false&hideText=false\" allowfullscreen></iframe>')"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "G5-XQWhivdiO",
        "cellView": "form"
      },
      "source": [
        "#@markdown Do you think we can solve the discrete XOR (only 4 possibilities) with only 2 hidden units?\r\n",
        "w3_min_xor = 'Select' #@param ['Select', 'Yes', 'No']"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ZMJZuSjmsEQV"
      },
      "source": [
        "---\n",
        "## Section 2.2: Universal Function Approximation Theorem"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "_uKiWhgZCSgY",
        "cellView": "form"
      },
      "source": [
        "#@title Video: Universal Approximation\n",
        "\n",
        "video = YouTubeVideo(id=\"XXXYxolMVdw\", width=854, height=480, fs=1)\n",
        "print(\"Video available at https://youtube.com/watch?v=\" + video.id)\n",
        "\n",
        "video"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "oS8AGBmqlRNZ"
      },
      "source": [
        "Check out [Lipschitz continuity](https://en.wikipedia.org/wiki/Lipschitz_continuity) for more information about the slide.\n",
        "\n",
        "So now that we saw multilayer perceptrons could solve the XOR problem. Building off this experiment, a natural question that comes to mind is - can we approximate any function using multilayer perceptrons?\n",
        "\n",
        " Universal Approximation theorem proves that we can! The intuition behind the theorem is that we can approximate any function sufficiently well, given a sufficient number of basis functions. These basis functions are present in the hidden layer of MLPs.  "
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "2j2qcuN8x32j"
      },
      "source": [
        "## Exercise 3: Function approximation with ReLU\r\n",
        "We learned that one hidden layer MLPs are enough to approximate any smooth function! Now let's manually fit a Sine function using ReLU activation. \r\n",
        "\r\n",
        "\r\n",
        "Instead of combining ReLUs with different slopes, we will approximate the sine function using a linear combination (a weighted sum) of ReLUs with slope 1. We need to determine the bias terms (which determines where the ReLU inflection point from 0 to linear occurs) and how to weight each ReLU. The idea is to set the weights iteratively so that the slope changes in the new sample's direction.\r\n",
        "\r\n",
        "First, we generate our \"training data\" from a sine function. These are the points we will use to learn how to approximate the function. We have 10 training data points so we will have 9 ReLUs (we don't need a ReLU for the last data point as we don't have anything to the right of it to model). \r\n",
        "\r\n",
        "We first need to figure out the bias term for each ReLU and compute the activation of each ReLU where:\r\n",
        "\r\n",
        "$$ y(x) = max(0, x + b) $$\r\n",
        "\r\n",
        "We then need to figure out the correct weights on each ReLU so the linear combination approximates the desired function."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "DVSelFbmFIE1"
      },
      "source": [
        "def approximate_function(x_train, y_train):\r\n",
        "\r\n",
        "    ####################################################################\r\n",
        "    # Fill in missing code below (...),\r\n",
        "    # then remove or comment the line below to test your function\r\n",
        "    raise NotImplementedError(\"Complete approximate_function!\")\r\n",
        "    ####################################################################\r\n",
        "\r\n",
        "    # Number of relus\r\n",
        "    n_relus = x_train.shape[0] - 1\r\n",
        "\r\n",
        "    # x axis points (more than x train)\r\n",
        "    x = torch.linspace(torch.min(x_train), torch.max(x_train), 1000)\r\n",
        "\r\n",
        "    ## COMPUTE RELU ACTIVATIONS\r\n",
        "\r\n",
        "    # First determine what bias terms should be for each of 9 ReLUs\r\n",
        "    b = ...\r\n",
        "\r\n",
        "    # Compute ReLU activations for each point along the x axis (x)\r\n",
        "    relu_acts = torch.zeros((n_relus, x.shape[0]))\r\n",
        "\r\n",
        "    for i_relu in range(n_relus):\r\n",
        "      relu_acts[i_relu, :] = torch.relu(x + b[i_relu])\r\n",
        "\r\n",
        "    ## COMBINE RELU ACTIVATIONS\r\n",
        "\r\n",
        "    # Set up weights for weighted sum of ReLUs\r\n",
        "    combination_weights = torch.zeros((n_relus, ))\r\n",
        "\r\n",
        "    # Figure out weights on each ReLU\r\n",
        "    prev_slope = 0\r\n",
        "    for i in range(n_relus):\r\n",
        "      delta_x = x_train[i+1] - x_train[i]\r\n",
        "      slope = (y_train[i+1] - y_train[i]) / delta_x \r\n",
        "      combination_weights[i] = ...\r\n",
        "      prev_slope = slope\r\n",
        "\r\n",
        "    # Get output of weighted sum of ReLU activations for every point along x axis\r\n",
        "    y_hat = ...\r\n",
        "\r\n",
        "    return y_hat, relu_acts, x\r\n",
        "\r\n",
        "# Make training data from sine function\r\n",
        "N_train = 10\r\n",
        "x_train = torch.linspace(0, 2*np.pi, N_train).view(-1, 1)\r\n",
        "y_train = torch.sin(x_train)\r\n",
        "\r\n",
        "\r\n",
        "### uncomment the lines below to test your function approximation\r\n",
        "# y_hat, relu_acts, x = approximate_function(x_train, y_train)\r\n",
        "# plot_function_approximation(x, relu_acts, y_hat)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "UIv9uGAD1BAr"
      },
      "source": [
        "# to_remove solution\r\n",
        "\r\n",
        "def approximate_function(x_train, y_train):\r\n",
        "\r\n",
        "    # Number of relus\r\n",
        "    n_relus = x_train.shape[0] - 1\r\n",
        "\r\n",
        "    # x axis points (more than x train)\r\n",
        "    x = torch.linspace(torch.min(x_train), torch.max(x_train), 1000)\r\n",
        "\r\n",
        "    ## COMPUTE RELU ACTIVATIONS\r\n",
        "\r\n",
        "    # First determine what bias terms should be for each of 9 ReLUs\r\n",
        "    b = -x_train[:9]\r\n",
        "\r\n",
        "    # Compute ReLU activations for each point along the x axis (x)\r\n",
        "    relu_acts = torch.zeros((n_relus, x.shape[0]))\r\n",
        "\r\n",
        "    for i_relu in range(n_relus):\r\n",
        "      relu_acts[i_relu, :] = torch.relu(x + b[i_relu])\r\n",
        "\r\n",
        "\r\n",
        "    ## COMBINE RELU ACTIVATIONS\r\n",
        "\r\n",
        "    # Set up weights for weighted sum of ReLUs\r\n",
        "    combination_weights = torch.zeros((n_relus, ))\r\n",
        "\r\n",
        "    # Figure out weights on each ReLU\r\n",
        "    prev_slope = 0\r\n",
        "    for i in range(n_relus):\r\n",
        "      delta_x = x_train[i+1] - x_train[i]\r\n",
        "      slope = (y_train[i+1] - y_train[i]) / delta_x \r\n",
        "      combination_weights[i] = slope - prev_slope\r\n",
        "      prev_slope = slope\r\n",
        "\r\n",
        "    # Get output of weighted sum of ReLU activations for every point along x axis\r\n",
        "    y_hat = combination_weights @ relu_acts \r\n",
        "\r\n",
        "    return y_hat, relu_acts, x\r\n",
        "\r\n",
        "# Make training data from sine function\r\n",
        "N_train = 10\r\n",
        "x_train = torch.linspace(0, 2*np.pi, N_train).view(-1, 1)\r\n",
        "y_train = torch.sin(x_train)\r\n",
        "\r\n",
        "\r\n",
        "### uncomment the lines below to test your function approximation\r\n",
        "y_hat, relu_acts, x = approximate_function(x_train, y_train)\r\n",
        "with plt.xkcd():\r\n",
        "  plot_function_approximation(x, relu_acts, y_hat)\r\n"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "1b37WDpWIUNU"
      },
      "source": [
        "---\r\n",
        "# Section 3: MLPs in Pytorch"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "it4t-UVzJeop",
        "cellView": "form"
      },
      "source": [
        "#@title Video: Making Multi-Layer Perceptrons\n",
        "\n",
        "try: t3;\n",
        "except NameError: t3=time.time()\n",
        "\n",
        "video = YouTubeVideo(id=\"bAhrg8Z8_r8\", width=854, height=480, fs=1)\n",
        "print(\"Video available at https://youtube.com/watch?v=\" + video.id)\n",
        "\n",
        "video"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "5XadSdjL2qWj"
      },
      "source": [
        "In the previous segment, we implemented a function to approximate any smooth function using MLPs. We saw that using Lipschitz continuity; we can prove that our approximation is mathematically correct. MLPs are fascinating, but before we get into the details on designing them, let's familiarize ourselves with some basic terminology of MLPs- layer, neuron, depth, width, weight, bias, and activation function. Armed with these ideas, we can now design an MLP given its input, hidden layers, and output size."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "j_TpUE0cHW1K"
      },
      "source": [
        "## Exercise 4: Implement a general-purpose MLP in Pytorch\r\n",
        "The objective is to design an MLP with these properties:\r\n",
        "* works with any input (1D, 2D, etc.)\r\n",
        "* construct any number of given hidden layers using ModuleList\r\n",
        "* use the same given activation function in all hidden layers"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "CUsq7SdqHx8a"
      },
      "source": [
        "class Net(nn.Module):\r\n",
        "    def __init__(self, actv, num_inputs, hidden_units, num_outputs):\r\n",
        "        super(Net, self).__init__()\r\n",
        "\r\n",
        "        # Assign activation function (exec allows us to assign function from string)\r\n",
        "        exec('self.actv = nn.%s'%actv)  \r\n",
        "\r\n",
        "        # Initialize layers of MLP\r\n",
        "        self.layers = nn.ModuleList()\r\n",
        "\r\n",
        "        ####################################################################\r\n",
        "        # Fill in missing code below (...),\r\n",
        "        # then remove or comment the line below to test your function\r\n",
        "        raise NotImplementedError(\"Create MLP Layers\")\r\n",
        "        ####################################################################\r\n",
        "\r\n",
        "        # Loop over layers and create each one\r\n",
        "        for i in range(len(hidden_units)):\r\n",
        "          next_num_inputs = hidden_units[i] \r\n",
        "          self.layers += ...\r\n",
        "          num_inputs = next_num_inputs\r\n",
        "\r\n",
        "        # Create final layer\r\n",
        "        self.out = nn.Linear(num_inputs, num_outputs)\r\n",
        "\r\n",
        "    def forward(self, x):\r\n",
        "\r\n",
        "        ####################################################################\r\n",
        "        # Fill in missing code below (...),\r\n",
        "        # then remove or comment the line below to test your function\r\n",
        "        raise NotImplementedError(\"Calculate the forward pass\")\r\n",
        "        ####################################################################\r\n",
        "\r\n",
        "        # Flatten inputs to 2D (if more than that)\r\n",
        "        x = ...\r\n",
        "\r\n",
        "        # Get activations of each layer\r\n",
        "        for layer in self.layers:\r\n",
        "          x = ...\r\n",
        "\r\n",
        "        # Get outputs\r\n",
        "        x = self.out(x) \r\n",
        "\r\n",
        "        return x\r\n",
        "\r\n",
        "### Uncomment below to create network and test it on input\r\n",
        "# net = Net(actv='LeakyReLU(0.1)',\r\n",
        "#     num_inputs = 2,\r\n",
        "#     hidden_units = [100, 10, 5],\r\n",
        "#     num_outputs = 1)\r\n",
        "\r\n",
        "# input = torch.zeros((100, 2))\r\n",
        "# y = net(input)\r\n",
        "# print(f'The output shape is {y.shape} for an input of shape {input.shape}')"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "12yuLUsp3v0J"
      },
      "source": [
        "# to_remove solution\r\n",
        "\r\n",
        "class Net(nn.Module):\r\n",
        "    def __init__(self, actv, num_inputs, hidden_units, num_outputs):\r\n",
        "        super(Net, self).__init__()\r\n",
        "\r\n",
        "        # Assign activation function (exec allows us to assign function from string)\r\n",
        "        exec('self.actv = nn.%s'%actv)  \r\n",
        "\r\n",
        "        # Initialize layers of MLP\r\n",
        "        self.layers = nn.ModuleList()\r\n",
        "\r\n",
        "        # Loop over layers and create each one\r\n",
        "        for i in range(len(hidden_units)):\r\n",
        "          next_num_inputs = hidden_units[i] \r\n",
        "          self.layers += [nn.Linear(num_inputs, next_num_inputs)]  \r\n",
        "          num_inputs = next_num_inputs\r\n",
        "\r\n",
        "        # Create final layer\r\n",
        "        self.out = nn.Linear(num_inputs, num_outputs)\r\n",
        "\r\n",
        "    def forward(self, x):\r\n",
        "\r\n",
        "        # Flatten inputs to 2D (if more than that)\r\n",
        "        x = x.view(x.shape[0], -1)  \r\n",
        "\r\n",
        "        # Get activations of each layer\r\n",
        "        for layer in self.layers:\r\n",
        "          x = self.actv(layer(x))  \r\n",
        "\r\n",
        "        # Get outputs\r\n",
        "        x = self.out(x) \r\n",
        "\r\n",
        "        return x\r\n",
        "\r\n",
        "### Uncomment below to create network and test it on input\r\n",
        "net = Net(actv='LeakyReLU(0.1)',\r\n",
        "    num_inputs = 2,\r\n",
        "    hidden_units = [100, 10, 5],\r\n",
        "    num_outputs = 1)\r\n",
        "\r\n",
        "input = torch.zeros((100, 2))\r\n",
        "y = net(input)\r\n",
        "print(f'The output shape is {y.shape} for an input of shape {input.shape}')"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "vPWedinyGHiY"
      },
      "source": [
        "## Section 3.2: ReLU in practice\r\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "OdQ08YiiGk06",
        "cellView": "form"
      },
      "source": [
        "#@title Video: Implementing ReLU\n",
        "\n",
        "video = YouTubeVideo(id=\"E8c1my2S4E8\", width=854, height=480, fs=1)\n",
        "print(\"Video available at https://youtube.com/watch?v=\" + video.id)\n",
        "\n",
        "video"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "dyOmkogLKuDt"
      },
      "source": [
        "## Exercise 5: Benchmark various ReLU implementations\r\n",
        "Implement and benchmark at least three different ReLU implementations. Use `%timeit` with test number 10 and repeat number 3. Which then takes an average of 10 runs and repeats 3 times, and reports the lowest (best) average time.\r\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "7jzG8GWvKtLt"
      },
      "source": [
        "x = torch.rand((10000, 10000)).to(dev) - 0.5\r\n",
        "print(\"Pytorch : \", end='')\r\n",
        "%timeit -n10 -r3 torch.relu(x)\r\n",
        "print('-----------------------------------------------')\r\n",
        "\r\n",
        "print(\"First: \", end='')\r\n",
        "%timeit -n10 -r3 ...  \r\n",
        "print('-----------------------------------------------')\r\n",
        "\r\n",
        "print(\"Second: \", end='')\r\n",
        "%timeit -n10 -r3 ...\r\n",
        "print('-----------------------------------------------')\r\n",
        "\r\n",
        "print(\"Third: \", end='')\r\n",
        "%timeit -n10 -r3 ...\r\n",
        "print('-----------------------------------------------')"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "RFtxCUou4aKY"
      },
      "source": [
        "# to_remove solution\r\n",
        "\r\n",
        "x = torch.rand((10000, 10000)).to(dev) - 0.5\r\n",
        "print(\"Pytorch : \", end='')\r\n",
        "%timeit -n10 -r3 torch.relu(x)\r\n",
        "print('-----------------------------------------------')\r\n",
        "\r\n",
        "print(\"First: \", end='')\r\n",
        "%timeit -n10 -r3 torch.max(x, torch.zeros_like(x))   \r\n",
        "print('-----------------------------------------------')\r\n",
        "\r\n",
        "print(\"Second: \", end='')\r\n",
        "%timeit -n10 -r3 x * (x > 0)   \r\n",
        "print('-----------------------------------------------')\r\n",
        "\r\n",
        "print(\"Third: \", end='')\r\n",
        "%timeit -n10 -r3 x[x < 0] = 0  \r\n",
        "print('-----------------------------------------------')\r\n",
        "\r\n",
        "print(\"Forth: \", end='')\r\n",
        "%timeit -n10 -r3 (torch.abs(x) + x) / 2    "
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "7VDVqxxaJlz8"
      },
      "source": [
        "## Section 3.3: Classification with MLPs"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "gXlMu7ouaKrb",
        "cellView": "form"
      },
      "source": [
        "#@title Video: Classification and CrossEntropy\n",
        "\n",
        "video = YouTubeVideo(id=\"bxzDe2pifKU\", width=854, height=480, fs=1)\n",
        "print(\"Video available at https://youtube.com/watch?v=\" + video.id)\n",
        "\n",
        "video"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "meZhRHa14_KM"
      },
      "source": [
        "Two potential loss functions we could use out of the box for multi-class classification are:\r\n",
        "* CrossEntropyLoss:\r\n",
        "This criterion expects a class index in the range $[0, C-1]$ as the target for each value of a $1D$ tensor of size minibatch. There are other optional parameters like class weights and class ignores. Check the documentation here for more detail. Then in the simplest case, it calculates this,\r\n",
        "\r\n",
        "$$\r\n",
        "\\operatorname{loss}(x, \\text { class })=-\\log \\left(\\frac{\\exp (x[\\text { class }])}{\\sum_{j} \\exp (x[j])}\\right)=-x[\\text { class }]+\\log \\left(\\sum_{j} \\exp (x[j])\\right)\r\n",
        "$$\r\n",
        "\r\n",
        "* MultiMarginLoss:\r\n",
        "The loss corresponding to class j is calculated as follows,\r\n",
        "$$\r\n",
        "l_j(x, y)=\\sum_{j\\neq y} \\max (0, \\operatorname{margin}-x[y]+x[j])\r\n",
        "$$\r\n",
        "Then it is averaged over all the class elements and all the mini-batch samples.\r\n",
        "\r\n",
        "Before we could start optimizing these loss functions, we need a dataset!"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "aCvA5etP2GOy"
      },
      "source": [
        "## Exercise 6: Simulate a Spiral classification dataset\r\n",
        "Let's turn this fancy-looking equation into a classification dataset"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "oTdmdB6oD0KU"
      },
      "source": [
        "$$\r\n",
        "\\begin{array}{c}\r\n",
        "X_{k}(t)=t\\left(\\begin{array}{c}\r\n",
        "\\sin \\left[\\frac{2 \\pi}{K}\\left(2 t+k-1\\right)\\right]+\\mathcal{N}\\left(0, \\sigma^{2}\\right) \\\\\r\n",
        "\\cos \\left[\\frac{2 \\pi}{K}\\left(2 t+k-1\\right)\\right]+\\mathcal{N}\\left(0, \\sigma^{2}\\right) \r\n",
        "\\end{array}\\right)\r\n",
        "\\end{array}, \\quad 0 \\leq t \\leq 1, \\quad k=1, \\ldots, K\r\n",
        "$$"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "utfWXn81FHT4"
      },
      "source": [
        "def create_spiral_dataset(K, sigma, N):\r\n",
        "\r\n",
        "    # Initialize t, X, y\r\n",
        "    t = torch.linspace(0, 1, N)\r\n",
        "    X = torch.zeros(K*N, 2)\r\n",
        "    y = torch.zeros(K*N)\r\n",
        "\r\n",
        "    ####################################################################\r\n",
        "    # Fill in missing code below (...),\r\n",
        "    # then remove or comment the line below to test your function\r\n",
        "    raise NotImplementedError(\"Create spiral data\")\r\n",
        "    ####################################################################\r\n",
        "\r\n",
        "    # Create data\r\n",
        "    for k in range(K):\r\n",
        "      X[k*N:(k+1)*N, 0] = ...  \r\n",
        "      X[k*N:(k+1)*N, 1] = ...   \r\n",
        "      y[k*N:(k+1)*N] = ...   \r\n",
        "\r\n",
        "    return X, y\r\n",
        "\r\n",
        "# Set parameters\r\n",
        "K = 4\r\n",
        "sigma = 0.5\r\n",
        "N = 1000\r\n",
        " \r\n",
        "### Uncomment below to visualize data when done\r\n",
        "# X, y = create_spiral_dataset(K, sigma, N)\r\n",
        "# plt.scatter(X[:, 0], X[:, 1], c = y)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "WA7lYMIW6c6e"
      },
      "source": [
        "# to_remove solution\r\n",
        "def create_spiral_dataset(K, sigma, N):\r\n",
        "\r\n",
        "    # Initialize t, X, y\r\n",
        "    t = torch.linspace(0, 1, N)\r\n",
        "    X = torch.zeros(K*N, 2)\r\n",
        "    y = torch.zeros(K*N)\r\n",
        "\r\n",
        "    # Create data\r\n",
        "    for k in range(K):\r\n",
        "      X[k*N:(k+1)*N, 0] = t*(torch.sin(2*np.pi/K*(2*t+k)) + sigma*torch.randn(N))   \r\n",
        "      X[k*N:(k+1)*N, 1] = t*(torch.cos(2*np.pi/K*(2*t+k)) + sigma*torch.randn(N))   \r\n",
        "      y[k*N:(k+1)*N] = k   \r\n",
        "\r\n",
        "    return X, y\r\n",
        "\r\n",
        "# Set parameters\r\n",
        "K = 4\r\n",
        "sigma = 0.16\r\n",
        "N = 1000\r\n",
        " \r\n",
        "### Uncomment below to visualize data when done\r\n",
        "X, y = create_spiral_dataset(K, sigma, N)\r\n",
        "with plt.xkcd():\r\n",
        "  plt.scatter(X[:, 0], X[:, 1], c = y)\r\n",
        "  plt.show()"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ZUhAlD-XcbTa"
      },
      "source": [
        "## Section 3.4: Training and Evaluation"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "eY2VN0MfcSPN",
        "cellView": "form"
      },
      "source": [
        "#@title Video: Cross-Validation\n",
        "\n",
        "video = YouTubeVideo(id=\"U6BFrVCLsWU\", width=854, height=480, fs=1)\n",
        "print(\"Video available at https://youtube.com/watch?v=\" + video.id)\n",
        "\n",
        "video"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "S3RZa-charD2"
      },
      "source": [
        "## Exercise 7: Implement it for a classfication task\r\n",
        "Steps to follow: \r\n",
        "  * Dataset shuffle\r\n",
        "  * Train/Test split\r\n",
        "  * Dataloader definition\r\n",
        "  * Training and Evaluation"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "1NCS2dEqPfiA"
      },
      "source": [
        "def shuffle_and_split_data(X, y):\r\n",
        "\r\n",
        "  ####################################################################\r\n",
        "  # Fill in missing code below (...),\r\n",
        "  # then remove or comment the line below to test your function\r\n",
        "  raise NotImplementedError(\"Shuffle & split data\")\r\n",
        "  ####################################################################\r\n",
        "\r\n",
        "  # Number of samples\r\n",
        "  N = X.shape[0]\r\n",
        "\r\n",
        "  # Shuffle data\r\n",
        "  shuffled_indices = ...  # get indices to shuffle data\r\n",
        "  X = X[shuffled_indices]\r\n",
        "  y = y[shuffled_indices]\r\n",
        "\r\n",
        "  # Split data into train/test\r\n",
        "  test_size = ...  # assign size of test data\r\n",
        "  X_test = X[:test_size]\r\n",
        "  y_test = y[:test_size]\r\n",
        "  X_train = X[test_size:]\r\n",
        "  y_train = y[test_size:]\r\n",
        "\r\n",
        "  return X_test, y_test, X_train, y_train\r\n",
        "\r\n",
        "\r\n",
        "### Uncomment below to test your function\r\n",
        "# X_test, y_test, X_train, y_train = shuffle_and_split_data(X, y)\r\n",
        "# plt.scatter(X_test[:, 0], X_test[:, 1], c=y_test)\r\n",
        "# plt.title('Test data')"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "ymb3r0pM6665"
      },
      "source": [
        "# to_remove solution\r\n",
        "def shuffle_and_split_data(X, y):\r\n",
        "\r\n",
        "  # Number of samples\r\n",
        "  N = X.shape[0]\r\n",
        "\r\n",
        "  # Shuffle data\r\n",
        "  shuffled_indices = torch.randperm(N)   # get indices to shuffle data\r\n",
        "  X = X[shuffled_indices]\r\n",
        "  y = y[shuffled_indices]\r\n",
        "\r\n",
        "  # Split data into train/test\r\n",
        "  test_size = int(0.2*N)    # assign size of test data\r\n",
        "  X_test = X[:test_size]\r\n",
        "  y_test = y[:test_size]\r\n",
        "  X_train = X[test_size:]\r\n",
        "  y_train = y[test_size:]\r\n",
        "\r\n",
        "  return X_test, y_test, X_train, y_train\r\n",
        "\r\n",
        "\r\n",
        "\r\n",
        "X_test, y_test, X_train, y_train = shuffle_and_split_data(X, y)\r\n",
        "with plt.xkcd():\r\n",
        "  plt.scatter(X_test[:, 0], X_test[:, 1], c=y_test)\r\n",
        "  plt.title('Test data')\r\n",
        "  plt.show()"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "TMYnhid1bCyq"
      },
      "source": [
        "And we need to make a Pytorch data loader out of it. Data loading in PyTorch can be separated in 2 parts:\r\n",
        "* Data must be wrapped on a Dataset parent class where the methods __getitem__ and __len__ must be overrided. Not that at this point the data is not loaded on memory. PyTorch will only load what is needed to the memory. Here `TensorDataset` does this for us directly.\r\n",
        "* Use a Dataloader that will actually read the data in batches and put into memory. Also, the option of `num_workers > 0` allows multithreading, which prepares multiple batches in the queue to speed things up."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "cPJNQtOOgC3E"
      },
      "source": [
        "batch_size = 128\r\n",
        "test_data = TensorDataset(X_test, y_test)\r\n",
        "test_loader = DataLoader(test_data, batch_size=batch_size,\r\n",
        "                         shuffle=False, num_workers=0)\r\n",
        "\r\n",
        "train_data = TensorDataset(X_train, y_train)\r\n",
        "train_loader = DataLoader(train_data, batch_size=batch_size, drop_last=True,\r\n",
        "                        shuffle=True, num_workers=0, worker_init_fn=seed_worker)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ZyRvSvQ8AJzk"
      },
      "source": [
        "Let's write a general-purpose training and evaluation code and keep it in our pocket for next tutorial as well. So make sure you review it to see what it does.\r\n",
        "\r\n",
        "Note that `model.train()` tells your model that you are training the model. So effectively layers like dropout, batchnorm etc. which behave different on the train and test procedures know what is going on and hence can behave accordingly. And to turn off training mode we set `model.eval()`"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "iW95_EQp1QHx"
      },
      "source": [
        "def train_test_classification(net, criterion, optimizer,\r\n",
        "                              train_loader, test_loader,\r\n",
        "                              num_epochs=1, verbose=True, \r\n",
        "                              training_plot=False):\r\n",
        "  if verbose:\r\n",
        "    progress_bar = display(progress(0, 0, num_epochs), display_id=True)\r\n",
        "\r\n",
        "  net.train()\r\n",
        "  training_losses = []\r\n",
        "  for epoch in range(num_epochs):  # loop over the dataset multiple times\r\n",
        "      running_loss = 0.0\r\n",
        "      for i, data in enumerate(train_loader, 0):\r\n",
        "          # get the inputs; data is a list of [inputs, labels]\r\n",
        "          inputs, labels = data\r\n",
        "          inputs = inputs.to(dev).float()\r\n",
        "          labels = labels.to(dev).long()\r\n",
        "\r\n",
        "          # zero the parameter gradients\r\n",
        "          optimizer.zero_grad()\r\n",
        "\r\n",
        "          # forward + backward + optimize\r\n",
        "          outputs = net(inputs)\r\n",
        "\r\n",
        "          loss = criterion(outputs, labels)\r\n",
        "          loss.backward()\r\n",
        "          optimizer.step()\r\n",
        "\r\n",
        "          # print statistics\r\n",
        "          if verbose:\r\n",
        "            training_losses += [loss.item()]\r\n",
        "            running_loss += loss.item()\r\n",
        "            if i % 10 == 9:    # update every 10 mini-batches\r\n",
        "                progress_bar.update(progress(epoch+1, running_loss / 10, num_epochs))\r\n",
        "                running_loss = 0.0\r\n",
        "\r\n",
        "  net.eval()\r\n",
        "  def test(data_loader):\r\n",
        "    correct = 0\r\n",
        "    total = 0\r\n",
        "    for data in data_loader:\r\n",
        "        inputs, labels = data\r\n",
        "        inputs = inputs.to(dev).float()\r\n",
        "        labels = labels.to(dev).long()\r\n",
        "\r\n",
        "        outputs = net(inputs)\r\n",
        "        _, predicted = torch.max(outputs, 1)\r\n",
        "        total += labels.size(0)\r\n",
        "        correct += (predicted == labels).sum().item()\r\n",
        "\r\n",
        "    acc = 100 * correct / total\r\n",
        "    return total, acc\r\n",
        "\r\n",
        "  train_total, train_acc = test(train_loader)\r\n",
        "  test_total, test_acc = test(test_loader)\r\n",
        "\r\n",
        "  if verbose:\r\n",
        "    print('Accuracy on the %d training samples: %0.2f %%' % (train_total, train_acc))\r\n",
        "    print('Accuracy on the %d testing samples: %0.2f %%' % (test_total, test_acc))\r\n",
        "\r\n",
        "  if training_plot:\r\n",
        "    plt.plot(training_losses)\r\n",
        "    plt.xlabel('Batch')\r\n",
        "    plt.ylabel('Training loss')\r\n",
        "    plt.show()\r\n",
        "  \r\n",
        "  return train_acc, test_acc"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "cellView": "form",
        "id": "UX-BvX_qBJl6"
      },
      "source": [
        "#@markdown Is it necessary to use `net.train()` and `net.eval()` for our MLP model? why?\r\n",
        "w3_why_two_modes = '' #@param {type:\"string\"}"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "c-g2TXrC_TOQ"
      },
      "source": [
        "Now let's put everything together and train your first deep-ish model!"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "iEGwbZrUfpZu"
      },
      "source": [
        "net = Net('ReLU()', X_train.shape[1], [128], K).to(dev)\r\n",
        "criterion = nn.CrossEntropyLoss()\r\n",
        "optimizer = optim.Adam(net.parameters(), lr=1e-3)\r\n",
        "num_epochs = 100\r\n",
        "_, _ = train_test_classification(net, criterion, optimizer, train_loader,\r\n",
        "                                 test_loader, num_epochs=num_epochs,\r\n",
        "                                 training_plot=True)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "6_HvMx0thX42"
      },
      "source": [
        "And finally, let's visualize the learned decision-map. We know you're probably running out of time, so we won't make you write code now! But make sure you have reviewed it since we'll start with another visualization technique next time."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "elWAOswugLak"
      },
      "source": [
        "def sample_grid(M=500, x_max = 2.0):\r\n",
        "  ii, jj = torch.meshgrid(torch.linspace(-x_max, x_max,M),\r\n",
        "                          torch.linspace(-x_max, x_max, M))\r\n",
        "  X_all = torch.cat([ii.unsqueeze(-1),\r\n",
        "                     jj.unsqueeze(-1)],\r\n",
        "                     dim=-1).view(-1, 2)\r\n",
        "  return X_all\r\n",
        "\r\n",
        "def plot_decision_map(X_all, y_pred, X_test, y_test, M=500, x_max = 2.0, eps = 1e-3):\r\n",
        "  decision_map = torch.argmax(y_pred, dim=1)\r\n",
        "\r\n",
        "  for i in range(len(X_test)):\r\n",
        "    indeces = (X_all[:, 0] - X_test[i, 0])**2 + (X_all[:, 1] - X_test[i, 1])**2 < eps\r\n",
        "    decision_map[indeces] = (K + y_test[i]).long()\r\n",
        "\r\n",
        "  decision_map = decision_map.view(M, M).cpu()\r\n",
        "  plt.imshow(decision_map, extent=[-x_max, x_max, -x_max, x_max], cmap='jet')\r\n",
        "  plt.plot()\r\n",
        "\r\n",
        "X_all = sample_grid()\r\n",
        "y_pred = net(X_all)\r\n",
        "plot_decision_map(X_all, y_pred, X_test, y_test)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "cellView": "form",
        "id": "VpVW1MjDCToL"
      },
      "source": [
        "#@markdown Do you think this model is performing well outside its training distribution? Why?\r\n",
        "w3_OoD = '' #@param {type:\"string\"}"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "31mPmt3bYS5H"
      },
      "source": [
        "# Submit your responses\r\n",
        "Please run the following cell and then press \"Submit\" so we can record your responses."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "-T0vbGaHYXtC",
        "cellView": "form"
      },
      "source": [
        "import time\r\n",
        "import numpy as np\r\n",
        "from IPython.display import IFrame\r\n",
        "#@markdown #Run Cell to Show Airtable Form\r\n",
        "#@markdown ##**Confirm your answers and then click \"Submit\"**\r\n",
        "\r\n",
        "def prefill_form(src, fields: dict):\r\n",
        "  '''\r\n",
        "  src: the original src url to embed the form\r\n",
        "  fields: a dictionary of field:value pairs,\r\n",
        "  e.g. {\"pennkey\": my_pennkey, \"location\": my_location}\r\n",
        "  '''\r\n",
        "  prefills = \"&\".join([\"prefill_%s=%s\"%(key, fields[key]) for key in fields])\r\n",
        "  src = src + prefills\r\n",
        "  src = \"+\".join(src.split(\" \"))\r\n",
        "  return src\r\n",
        "\r\n",
        "\r\n",
        "#autofill time if it is not present\r\n",
        "try: t0;\r\n",
        "except NameError: t0 = time.time()\r\n",
        "try: t1;\r\n",
        "except NameError: t1 = time.time()\r\n",
        "try: t2;\r\n",
        "except NameError: t2 = time.time()\r\n",
        "try: t3;\r\n",
        "except NameError: t3 = time.time()\r\n",
        "\r\n",
        "#autofill fields if they are not present\r\n",
        "#a missing pennkey and pod will result in an Airtable warning\r\n",
        "#which is easily fixed user-side.\r\n",
        "try: my_pennkey;\r\n",
        "except NameError: my_pennkey = \"\"\r\n",
        "\r\n",
        "try: my_pod;\r\n",
        "except NameError: my_pod = \"Select\"\r\n",
        "\r\n",
        "try: w2_upshot;\r\n",
        "except NameError: w2_upshot = \"\"\r\n",
        "\r\n",
        "try: w3_q;\r\n",
        "except NameError: w3_q = \"\"\r\n",
        "\r\n",
        "try: w3_min_xor;\r\n",
        "except NameError: w3_min_xor = \"\"\r\n",
        "\r\n",
        "try: w3_why_become_relu;\r\n",
        "except NameError: w3_why_become_relu = \"\"\r\n",
        "\r\n",
        "try: w3_why_two_modes;\r\n",
        "except NameError: w3_why_two_modes = \"\"\r\n",
        "\r\n",
        "try: w3_OoD;\r\n",
        "except NameError: w3_OoD = \"\"\r\n",
        "\r\n",
        "\r\n",
        "times = np.array([t1,t2,t3])-t0\r\n",
        "\r\n",
        "fields = {\"pennkey\": my_pennkey,\r\n",
        "          \"pod\": my_pod,\r\n",
        "          \"w2_upshot\":w2_upshot,\r\n",
        "          \"w3_q\": w3_q,\r\n",
        "          \"w3_why_become_relu\":w3_why_become_relu,\r\n",
        "          \"w3_min_xor\": w3_min_xor,\r\n",
        "          \"w3_why_two_modes\":w3_why_two_modes,\r\n",
        "          \"w3_OoD\":w3_OoD,\r\n",
        "          \"cumulative_times\": times}\r\n",
        "\r\n",
        "src = \"https://airtable.com/embed/shrO0aY7Sz8u8qY2H?\"\r\n",
        "\r\n",
        "#now instead of the original source url, we do: src = prefill_form(src, fields)\r\n",
        "display(IFrame(src = prefill_form(src, fields), width = 800, height = 400))"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "zhhpq4gvanhu"
      },
      "source": [
        "## Feedback\r\n",
        "How could this session have been better? How happy are you in your group? How do you feel right now?\r\n",
        "\r\n",
        "Feel free to use the embeded form below or use this link:\r\n",
        "<a target=\"_blank\" rel=\"noopener noreferrer\" href=\"https://airtable.com/shrNSJ5ECXhNhsYss\">https://airtable.com/shrNSJ5ECXhNhsYss</a>"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "cellView": "form",
        "id": "NQ3oX2LFa3AB"
      },
      "source": [
        "# @title Feedback form\r\n",
        "display(IFrame(src=\"https://airtable.com/embed/shrNSJ5ECXhNhsYss?backgroundColor=red\", width = 800, height = 400))"
      ],
      "execution_count": null,
      "outputs": []
    }
  ]
}